Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

treewide: support structuredAttrs in setup hooks #318614

Merged

Conversation

wolfgangwalther
Copy link
Contributor

@wolfgangwalther wolfgangwalther commented Jun 9, 2024

Description of changes

I tried to build a package with __structuredAttrs which uses meson, realized that meson setup hooks don't support structuredAttrs, yet, and then... this happened.

The first 5 commits make the helpers in stdenv related to structuredAttrs a bit better re-usable. The remaining commits are using the "new" concatTo and prependToVar / appendToVar to fix many setup hooks.

The main goal here is to support structuredAttrs in those setup hooks without having any "if structuredAttrs then ..." conditionals except for a few in stdenv. I think this works pretty well.

There's a lot of changes in here, but many of them should be very easy to review. Most of them were tested by building a package using the related hooks: Once with structuredAttrs on master, which would fail, then with and without structuredAttrs on this branch - both would pass.

In any case, I pushed all those commits mostly to show how those helpers in stdenv would be used across the different hooks. I am happy to split up the PR into smaller pieces later and would welcome feedback especially on the first 5 commits.

The problem in 99% of those hooks is that $xxxFlags is used without [@], which will always just return the first array item when using structuredAttrs. However a simple change to $xxxFlags[@] isn't going to fly, if whitespace should be treated properly.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@wolfgangwalther
Copy link
Contributor Author

Hm, so @AndersonTorres is listed as maintainer for scons, but was not mentioned by ofborg. I assume that's because I only change the setup-hook.sh file and not a nix file.

Do I need to mention all maintainers manually in this case? I assumed ofborg would also work by commit prefix scons:, but that didn't add the mention either.

@wolfgangwalther
Copy link
Contributor Author

Rebased to resolve merge conflicts. Anyone up for a review of the first five stdenv: commits for the general base?

@philiptaron
Copy link
Contributor

I did a full build and test of the stdenv and everything checks out. I think the printf suggestion can be a separate PR.

I raise the motion that we merge this PR.

@SomeoneSerge
Copy link
Contributor

We'll have to look out to ensure nobody else starts to depend on the stdout-based concatStringsSep in the meantime, but I think that's manageable:)

@SomeoneSerge SomeoneSerge merged commit ccaaa9c into NixOS:staging Aug 13, 2024
26 checks passed
@emilazy
Copy link
Member

emilazy commented Aug 13, 2024

Really great to see this land! Looking forward to the follow‐up for the other hooks. Let’s all do our best to get structured attributes on by default as soon as we can.

@trofi
Copy link
Contributor

trofi commented Aug 14, 2024

Bisect says the bfd97a6 stdenv: make _accumFlagsArray independent of structuredAttrs changed quoting rules for some variables. For example sudo is now broken in staging as:

$ nix build --no-link -f. -L sudo
...
sudo> ./configure: interpreter directive changed from "#! /bin/sh" to "/nix/store/k5i08diz6c0bvklbzdp35rpprrnml9bh-bash-5.2p32/bin/sh"                               
sudo> configure flags: --disable-static --prefix=/nix/store/8kqimiqs90p0vlvm43ia2v46wc040gbw-sudo-1.9.15p5 --with-env-editor --with-editor=/run/current-system/sw/bin
/nano --with-rundir=/run/sudo --with-vardir=/var/db/sudo --with-logpath=/var/log/sudo.log --with-iologdir=/var/log/sudo-io --with-sendmail=/run/wrappers/bin/sendmail
 --enable-tmpfiles.d=no password for %p:                                                                                                                             
sudo> configure: WARNING: you should use --build, --host, --target                                                                                                   
sudo> configure: WARNING: you should use --build, --host, --target                                                                                                   
sudo> configure: WARNING: you should use --build, --host, --target                                                                                                   
sudo> configure: WARNING: invalid host type: %p:        
...
sudo> checking for password-ar... ar
sudo> checking for password-ranlib... ranlib
sudo> checking build system type... Invalid configuration 'password': machine 'password-unknown' not recognized
sudo> configure: error: /nix/store/k5i08diz6c0bvklbzdp35rpprrnml9bh-bash-5.2p32/bin/bash ./scripts/config.sub password failed

This happenens because sudo has this snippet:

  configureFlagsArray = [
    "--with-passprompt=[sudo] password for %p: " # intentional trailing space
  ];

It used to be passed as a single string, but now it gets expanded to an argument list.

@emilazy
Copy link
Member

emilazy commented Aug 14, 2024

I guess we just need to restore the special handling of *Array? I’ll try to take a look in the next day or so if nobody beats me to it.

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Aug 14, 2024

It used to be passed as a single string, but now it gets expanded to an argument list.

Most peculiar, I'm surprised this used to work because this does not seem to be a __structuredAttrs derivation yet it sets *Array as a nix variable...

EDIT: it probably worked because *Array was used with "${...[@]}" unconditionally of it being an actual array, so maybe bash saw it as "a single element-array that wasn't declared as an array" or something - I'm not a bash expert, but I think we should simply enable __structuredAttrs:)

@emilazy
Copy link
Member

emilazy commented Aug 14, 2024

I guess just toggling on __structuredAttrs would be nice, but this is going to break out‐of‐tree packages. We probably need to fix it.

@SomeoneSerge
Copy link
Contributor

I guess just toggling on __structuredAttrs would be nice, but this is going to break out‐of‐tree packages. We probably need to fix it.

👍🏻 Two separate matters: make sudo structuredAttrs, add a warning/error out-of-tree users when [[ name = *"Array" ]], possibly recovering the old behavior

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Aug 14, 2024

FWIW I think the way *Array variables are used in the manual is they are set on the bash side, e.g. in preConfigure

EDIT:

I’ll try to take a look in the next day or so if nobody beats me to it.

I'm probably free in a couple of hours

@philiptaron
Copy link
Contributor

With #334973 merged, I think we're safe for staging-next. Anyone disagree?

wolfgangwalther added a commit to wolfgangwalther/nixpkgs that referenced this pull request Aug 24, 2024
The previously used pattern was introduced in NixOS#318614, but technically
leaked the default flags into the global scope. While this would
probably not make much of a practical difference, making concatTo
support default values is a much cleaner approach.
wolfgangwalther added a commit to wolfgangwalther/nixpkgs that referenced this pull request Sep 8, 2024
cudaPackages already builts with structuredAttrs, but the cmakeFlags+=
pattern incorrectly appends the additional flags to the first array
argument with a space - which is now part of that argument itself since
NixOS#318614, which added support for structuredAttrs to cmake.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants